Inspiration
One day I was walking my dog, and it had started playing with another dog on the street. I quickly became friends with the other dog's owner, and it inspired me to create an application in XR to help address issues of loneliness by creating organic interactions in the wild.
What it does
We build a colocation AR app that allows up to ten users to interact with each other's pets. Users interact with pets by verbally talking to them in natural language like they would with a real pet. Pets understand users using an off-cloud Large Language Model to drive pet behaviors such as intelligent pathfinding based on semantic understanding of the environment (ie. a pet will understand what a chair is automatically and perform contextually appropriate actions such as jumping onto it).
How we built it
We built this app using the Vision Pro and Unity. We leverage an open source MIT-licensed multimodal agent framework, Estuary, to drive the agent behavior.
Challenges we ran into
Technical- lots of radio noise in the hacking hall especially since we're working with both multiplayer and a distributed computing platform (Estuary) over local area network. There were a lot of advanced spatial understanding features we needed to leverage which also meant we had to build each time we want to test rather than preview in editor. Localization- there is no support in Unity for any of Apple's AVP colocation features. Hence we have to start all users from the same starting point when launching the experience.
Accomplishments that we're proud of
Coming together as a team that has never worked together before and integrating a bunch of nascent features that had next to no tutorials nor documentation online!
What we learned
How to make a networked colocation AR application, how to prompt an LLM to drive an agent using function calls!
What's next for Paw Pals Connect
Complex pet to pet interactions, sophisticated pet personality development features, and pet customization!
Log in or sign up for Devpost to join the conversation.